Many common data systems capture data from multiple data sources. It is often desirable to have data from these multiple data sources stored on the same storage device or within the same storage system. These types of systems often capture and store large quantities of data over extended periods of time. These large quantities of data require large amounts of storage space and new data is often being acquired and stored on an ongoing basis. Consequently, the amount of available storage space can become a limiting factor.
A need to access or use the stored data may not arise for a significant period of time after it has been stored. Because it is difficult to know when a need for the data may arise, it is challenging to know how long to keep the data before it is deleted. As a result, the data is often retained until it must be deleted in order to make space available for recently acquired data. One approach is to delete or overwrite the oldest data as it should be the least likely to be needed in the future.
Data systems typically store data in files or some other type of file-based format. As long as space is available, it is often desirable to retain data as long as possible in case a later need for the data arises. Consequently, storage devices are often completely filled with data and the old data deleted or overwritten only when additional storage space is needed for new data.
One example of a data system like that described above is a video system used for security or surveillance applications. In these security and surveillance applications, multiple cameras or video sources are used to capture video of multiple scenes or locations. The video is often stored in a single storage system enabling a user to more easily work with the combined video, search video from multiple sources, link video from multiple sources, or perform other activities involving segments of video that may have been received from more than one video source.
The specific events, scenes, or incidents in the video that may be of interest are often not known ahead of time. Consequently, video is captured and stored over extended periods of time, if not continuously, in order to improve the chances of capturing unexpected, unplanned, or presently unknown incidents or occurrences. It is often not known when a need for the video may arise, so it is desirable to store the video and have it available for use as long as reasonably possible in case a need arises. The desire to have the video available as long as possible is counterbalanced by the limited storage space that may be available in the storage devices, systems, or locations.
Multiple methodologies and types of file systems are used for storing video in these types of applications. Video from each source is typically stored in its own file. If contiguous physical storage space is available on the storage device, video associated with a single file may be stored in a contiguous physical storage space as much as is possible. As files are deleted and added, scattered pockets of available storage space may become available, particularly if the added and deleted files are of different sizes. As a result, video making up a single file may be scattered across multiple physical locations on the storage device or storage devices, and the storage device becomes fragmented with the various files.
When video files are stored in the manner described above, a storage system that is receiving and storing video streams from multiple sources must constantly move between several physical locations to store those video streams, because the files associated with each of the video sources are in different physical locations of the device or devices.
Indexes, such as file allocation tables, must be used to keep track of where each file is located and where the next record in each of the streams should be stored with respect to each of the files. If a disk drive is being used to store the video, read and write heads of the disk drive must move to a different place on the disk as it cycles through the process of storing records or other discrete segments of video associated with each of the video sources. This can be limiting as the time to identify the next write location and reposition the head to that location may be very time consuming and may have to be performed many times per second for each data stream.
In addition, the problem of deleting old video in order to make room for new video on a storage device that is full suffers some of the same challenges. In order to keep old video as long as possible, old files are often not deleted until new video has arrived, or is about to arrive, for which the storage space is needed. Because the oldest data may be scattered around in various physical locations, the storage device or storage device controller must perform additional tasks or processes to determine which video is the oldest, where it is located, and take the necessary actions to move the head to that location to erase or overwrite the data. This process is time consuming and means that new data is further stored in non-sequential or non-contiguous physical locations on the storage device or devices.
In examples where data is stored in large files, it can be time consuming to retrieve the large files, remove an old portion of the file, and re-write the file to the storage device. If the old portion of the file is at the beginning portion of the file, a truncation process is slow and cumbersome as the disk drive must read the entire file, modify the file accordingly, and re-write the file back to the storage medium.
Overview
Methods, systems, and software are provided herein that allow a user to store and retrieve data records. In a first example, a method of storing data records is provided. The method includes initializing a shared write pointer to a first location in a shared memory of a data storage system, and receiving a plurality of data records transferred by a plurality of data sources for storage in the shared memory. The method also includes, beginning with a first of the plurality of data sources, writing in a cyclic sequence one of the plurality of data records for each of the plurality of data sources based on the shared write pointer, and incrementing the shared write pointer after each writing, and updating at least a header portion of the shared memory with the shared write pointer responsive to each writing.
In a second example, a data storage system is disclosed. The data storage system includes a plurality of data sources, a shared storage volume accessible to the plurality of data sources, and a storage controller. The storage controller is configured to receive a first data record, identify a source of said first data record to generate a source identifier, store on the shared storage volume source identifier identified in the first data record in an index, store on the shared storage volume the first data record at a first storage location indicated by a pointer, and increment the pointer after storing the first data record. The storage controller is also configured to receive a second data record, and store on the shared storage volume the second data record at a second storage location indicated by the incremented pointer.
In a third example, a computer readable medium is disclosed. The computer readable medium has instructions stored thereon, that when executed, perform the steps of receiving in a data storage system a first data record, identifying a source of said first data record to generate a source identifier, storing the source identifier identified in the first data record in an index, storing the first data record at a first storage location indicated by a pointer, and incrementing the pointer after storing the first data record. The computer readable medium has further instructions stored thereon, that when executed, perform the steps of receiving a second data record, and storing the second data record at a second storage location indicated by the incremented pointer.